feat(cm): mirror the critical-minerals workbook into CM_legacy - #867

Phase-1 staging mirror of the Earth MRI critical-minerals chemistry workbook (McLemoreMasterChem, compiled by V.T. McLemore et al., NMBGMR) into CM_* tables, plus a reconciliation report for the source-data owner. No transform into the Ocotillo data model yet.

What changed

Seven mirror tables, following the db/nma_legacy.py / db/nmw_legacy.py convention:

• db/cm_legacy.py — models, plus SOURCE_HEADER_BY_COLUMN and ANALYTE_UNITS
• alembic/versions/d4e5f6a7b8c9_cm_legacy_mirror_tables.py — migration, single head
• services/cm_legacy_mirror.py — loader, idempotent per sheet
• scripts/cm_reconciliation_report.py — builds the decision workbook
• oco load-critical-minerals-workbook --file <xlsx>
• docs/critical-minerals-legacy-mirror.md, CLAUDE.md pointer
• 51 tests across tests/test_cm_legacy.py and tests/test_cm_reconciliation_report.py

Why it looks like this

Three sheets, one table. ChemicalData and GIS are byte-identical in header text and order (118 columns each); QAQC is those columns minus MapSymbol/Pd/Pt with latitude/longitude capitalized. Only the layout differs, so one table behind a source_sheet discriminator beats three near-duplicate tables — and it makes reconciliation a self-join.

Rows are keyed on (source_sheet, source_row), not SAMPLE. Sample names repeat across 258 distinct names in the source, so SAMPLE cannot be the key. source_row is the 1-based Excel row number, which makes every mirror row traceable to a cell range in the delivered workbook and a reload idempotent per sheet.

Every column is a String. The workbook stores censored analyte values as text (1154 < values in Au alone), carries #VALUE! errors, and mixes real dates with year-only text. Any typed column drops cells; parsing value-plus-qualifier and casting dates belong to the transform.

Column names are derived, not copied. Sheet headers are spreadsheet labels, not SQL identifiers (Chem Lab File No., Depth/legnth (ft), H2O+), so unlike the NMA/NMW mirrors the source name cannot be reused verbatim. Analytes carry the unit the workbook declares for them (sio2_pct, au_ppb, as_ppm), which also keeps As and In off the Python and SQL keyword lists. SOURCE_HEADER_BY_COLUMN holds the exact header for all 118 columns.

The load asserts its layout. Header rows are declared per sheet (ChemicalData 2, GIS 1, QAQC 2); a moved header row or an unmapped column aborts the load with nothing written, rather than silently dropping cells. A revised workbook that changes shape needs a human and a migration.

Reviewer should know: reconciliation is deferred on purpose

GIS is not ChemicalData plus location data, and not a clean subset. It is a stale hand-maintained fork: same mixed datums, same ~876 rows with no latitude, so it adds no location information. Every GIS sample name exists in ChemicalData, but 1704 of the 4848 shared rows disagree in both directions:

Plus 18 sample rows appended to ChemicalData after GIS was last synced — all Pearce (2020), all with coordinates.

Neither sheet is authoritative, so both are mirrored in full and the merge is left to a per-column ruling by V.T. McLemore. A query reading only source_sheet='ChemicalData' silently drops 533 lab file numbers, 485 lab names and 170 FeO/Fe2O3 values. The loader warns whenever the two row counts diverge.

The reconciliation report

scripts/cm_reconciliation_report.py writes an 8-sheet decision workbook: ColumnDecisions (8), CellDifferences (2123), RowsOnlyInOneSheet (19), IntegritySummary (13), IntegrityDetail (4276), DetectionLimitSpread (64), DuplicateSampleNames (258). Every judgement column is blank with a dropdown.

Integrity findings from ChemicalData alone include 724 values impossible for their declared unit (F = 27700 in a % column), 259 Au detection limits that are ppm-scale in a ppb column, 300 non-numeric analyte tokens (bd, nd, tr, nr, ----, >2%), 159 Totals that disagree with the sum of the majors, and 64 of 84 analytes reported against more than three distinct detection limits.

Total is deliberately exempt from the unit check — it is a sum, so exceeding 100 is not itself a unit error; flagging it added 1519 false positives. Only ChemicalData is integrity-checked, since running the same checks over GIS would double every finding without adding information.

Verification

• Full suite green: 1070 passed, 81 skipped, 6 xpassed
• Migration DDL executed against the test database in a rolled-back transaction; all 7 tables match the model metadata column-for-column
• Loaded the delivered workbook end to end: 10,100 rows (ChemicalData 4867, GIS 4848, QAQC 8, DetectionLimits 61, References 81, MineralSystems 169, world 7, world_ref 4, metadata 24/9/22)
• 708 cells across 6 random rows compared against openpyxl with zero mismatches. Full float precision survives (101.83999999999999, not 101.84), censored values keep their qualifier, #VALUE! passes through, dates become ISO-8601 instead of Excel serials

Notes

• Nothing here reads or writes any existing table; the mirror is additive
• services/cm_legacy_mirror.py exposes cell_to_text, normalize_header and column_by_header publicly so the report reuses the mirror's exact cell rendering rather than reimplementing it
• Loading is manual (oco load-critical-minerals-workbook) — like the other legacy mirrors, there is no CD path for data loads

🤖 Generated with Claude Code